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Abstract Body 
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Background / Context: 

Description of prior research and its intellectual context. 


The School Improvement Grants (SIG) program received over $3 billion through the American 
Recovery and Reinvestment Act of 2009. Through grants to states, SIG focuses on turning 
around the nation’s persistently lowest- achieving schools using one of four school intervention 
models, with the aim of substantially improving student achievement. Consistent with the 
conference theme, SIG is designed to create sustained achievements over time. Rigorous 
evidence of the impact of this program on student outcomes has been limited to a study by Dee 
(2012) that focused on a single state (California). Dee used school-level data, an approach that 
conflates impacts on student achievement with impacts on student mobility. That is, the positive 
impacts found by Dee could be because SIG schools are more effective at educating children or 
because SIG schools simply attracted higher- achieving students. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 


Does receipt of SIG funding to implement a school intervention model have an impact on 
outcomes for low-performing schools? We will answer this question using a regression 
discontinuity design (RDD) that exploits cutoff values on the continuous variables used to define 
SIG eligibility tiers, comparing outcomes in schools that just met the eligibility criteria to 
outcomes in schools that just missed the eligibility cutoff (while controlling for the variable used 
to determine tier assignment). Using student-level data, we will assess whether SIG affected 
student mobility and will calculate “mobility-robust” impacts using students slated to attend 
study schools in the implementation year based on the school they attended the prior year. In 
essence, we will use the school attended the prior year as an “instrument” for school attendance 
during the implementation year. This approach yields impacts on student achievement that are 
not conflated with impacts on student mobility. For comparison, we will also calculate “place- 
based” impacts using students who attended study schools in the implementation year. 

Setting: 

Description of the research location. 

(May not be applicable for Methods submissions) 

Low-performing schools in 21 states and the District of Columbia (DC). 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

(May not be applicable for Methods submissions) 

We will focus on the effect of SIG awards made in 2010, when over $3 billion in awards were 
made to all 50 states and DC. Our sample was purposively selected to (1) include only those 
schools that can contribute to a valid RDD and (2) maximize the precision of impacts. The 
sample includes 1,100 low-performing schools in about 60 districts in 21 states and DC. Using 
estimates of fuzziness and sample size based on information gathered through a review of states’ 
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SIG application materials and conversations with state administrative staff, we calculated the 
minimum detectable effect size (MDES) corresponding to every opportunity to estimate an RDD 
impact in every state. We then ranked those opportunities and prioritized states and districts 
corresponding to the opportunities with lower MDES values. 

The characteristics of states in our sample do not differ significantly from states nationwide 
(insert Table 1 here). The districts in our sample differ from all districts in which SIG schools are 
located on students’ race and school location. Our sample districts have a higher percentage of 
non-Hispanic black students and are more likely to be in an urban area. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

(May not be applicable for Methods submissions) 

The treatment is defined as receiving SIG funds for implementing one of four school intervention 
models. States categorized low-performing schools into three eligibility tiers defined by ED. ED 
required each SIG-awarded school in Tier I or II to implement one of four school intervention 
models over the course of three school years (starting in 2010-2011). Each model prescribed 
specific practices (insert Exhibit 1 here). Schools in Tier III were permitted but not required to 
implement a model. Table 2 shows award amounts and the distribution of SIG grantees across 
tiers and models (insert Table 2 here). 

Significance / Novelty of study: 

Description of what is missing in previous work and the contribution the study makes. 


To our knowledge, this is the first large-scale RDD that tackles a host of methodological issues 
simultaneously (including multiple sites, multiple assignment variables, fuzziness, clustering of 
individuals within schools, and standard errors that take into account the bandwidth selection 
method). Although many of these issues are not new to the field, this study is the first to face 
them all at once. Because our analysis methods are based on simulation work and consultation 
with RDD experts, we think other education researchers will benefit from learning about them. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 


We will use an RDD to estimate the impact of SIG-funded intervention models on student 
outcomes, taking advantage of RDD opportunities created by ED rules about the prioritization of 
SIG funds, which use cutoffs on continuous school-level variables such as achievement and 
graduation rates. The definitions of SIG eligibility tiers are in Table 3 and the opportunities they 
create to estimate RDD impacts are in Table 4 (insert Tables 3 and 4 here). Schools in Tiers I and 
II form the treatment group; schools in Tier III and ineligible schools form the comparison 
group. Our analytic approach has the following features: 

Local linear impact estimation. We will estimate impacts within a bandwidth around the cutoff 
value, adjusting for the assignment variable (i.e., the variable used to assign schools to the 
intervention) using a linear functional form. The coefficient on the assignment variable will be 
estimated separately above and below the cutoff. 
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Bandwidth selection based on the Imbens & Kalyanaraman (2012) (IK) method. We will use 
the IK bandwidth selection method to estimate a single study-wide bandwidth. We will first 
standardize the outcome and assignment variables from each grade and RDD opportunity. 
Outcome variables will be centered at their means and assignment variables will be centered at 
their cutoff values. Both will be divided by their standard deviations. The standardized variables 
will be pooled. The IK method will be applied to the pooled data. Impacts will be estimated 
separately in each grade within the IK bandwidth that was selected using pooled data. We chose 
this method based on its performance relative to alternatives in simulations. First, we selected the 
IK method relative to cross-validation (Ludwig & Miller, 2007) because, in simulations, it 
produced estimated impacts with less bias (insert Table 5 here). Second, our simulations 
suggested that calculating an IK bandwidth using pooled data (which we call “study-wide IK”) 
yielded smaller standard errors than either (1) applying the IK method to each grade and RDD 
opportunity separately (which we call “local IK”) or (2) modifying the IK method so that sample 
sizes and densities are estimated locally while derivatives and conditional variances are 
estimated using pooled data (a method suggested by Imbens which we call “hybrid IK”) (insert 
Table 7 here). 

Both single and multiple assignment variable RDD. Some grades have a single assignment 
variable (either average achievement or graduation rate) and others have two assignment 
variables (both average achievement and graduation rate). For grades with two assignment 
variables we will calculate impacts using both the “Frontier” and “Fuzzy Frontier” methods 
(Reardon and Robinson, 2010). When calculating the weighted average of the impacts from the 
two assignment variables, we will adjust the standard errors to account for overlap of schools 
across the two impact estimates. 

Aggregation of impacts across grades. For each outcome measure, we will estimate impacts 
separately for each grade so that the relationship between the assignment variable and outcome is 
modeled separately across grades. We will then calculate an aggregate impact that is a sample 
size weighted average of the grade- specific impacts, where the sample size is the number of 
students in study schools within the bandwidth. In calculating the weighted average of these 
impacts and the corresponding standard errors, we will account for impact covariance due to the 
overlapping samples between impacts. 

Covariates. We will include pre-intervention test scores and an indicator of eligibility for free or 
reduced-price lunch as covariates to increase precision. We will include site (i.e., state) dummies 
to control for variation across sites in the relationship between the assignment variable and the 
outcome. 

Bootstrapped standard errors. We will estimate standard errors through residual bootstrapping 
to account for clustering of students within unique values of the assignment variable (Lee and 
Card 2008), fuzziness (described below), correlations among grade impacts due to overlapping 
samples (for example, schools that meet the requirement for inclusion in more than one 
eligibility tier), and variance introduced by the bandwidth selection method. 
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Fuzzy RDD impacts. This study is a “fuzzy” RDD, meaning that not all schools below the cutoff 
implemented a SIG-funded intervention model, and some schools above the cutoff did. We will 
calculate the impact of SIG-funded intervention models on outcomes by estimating the local 
average treatment effect (LATE). The LATE equals the RDD impact on the outcome divided by 
the RDD impact on the proportion of schools implementing a SIG-funded intervention model. 

Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 


We conducted Monte Carlo simulations to show that the analytic approach described above 
yields unbiased and efficient impact estimates, and to compare the performance of the various 
bandwidth selection methods described above. The process used to generate simulated data 
mimics study data observed from the year before SIG had an opportunity to affect outcomes. 
Thus, the simulations were conducted under the null hypothesis of no true impact, using a data 
generating process (DGP) that mimics the relationships between outcomes and assignment 
variables observed in our actual data. 

Research Design: 

Description of the research design. 

(May not be applicable for Methods submissions) 

See the “Statistical, Measurement, or Econometric Model” section. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for Methods submissions) 

Assignment variable values for each school were collected from states. Outcomes — student 
standardized test scores on state assessments, high school graduation, and college enrollment (a 
longer-term outcome consistent with the conference theme) — were collected from student-level 
administrative data maintained by states and districts. We will examine each outcome for the 
2010-2011, 2011-2012, and 2012-2013 school years. Our “mobility- robust” method described 
above calculates impacts based on following students over time, including any transitions they 
make from one school level to the next (e.g., elementary to secondary). 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for Methods submissions) 

Not applicable, bindings will be presented in a future IBS report. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 


Not applicable. Conclusions will be presented in a future IBS report. 
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Appendix B. Tables and Figures 

Not included in page count. 


Table 1. Baseline Characteristics of the State and District Samples 



Study States 

All States 

Study Districts 

Districts in the 
U.S. With at Least 
One School 
Implementing a 
SIG-Funded 
Intervention Model 

Average Percentage of Students by 
Racial/Ethnic Category 

White, non-Hispanic 

55.3 

61.8 

19.5* 

33.4 

Black, non-Hispanic 

19.5 

15.8 

38.7* 

30.3 

Hispanic 

18.3 

13.7 

32.0 

25.8 

Asian 

3.8 

4.6 

3.3 

2.5 

Other 

3.1 

4.1 

6.5 

8.0 

Average Percentage of Students Eligible for 

Free or Reduced-Price Lunch 

48.0 

45.5 

72.4 

68.1 

Percentage of Schools That Are Title 1 

Eligible 

68.1 

67.8 

81.4 

83.0 

Percentage of Schools by Location 

Urban 

30.0 

23.3 

68.2* 

37.7 

Suburban 

25.7 

22.5 

17.3 

20.0 

Town or rural 

44.3 

54.2 

14.5* 

42.3 

Number of States or Districts 

22 

51 

60 

610 


Sources: Common Core of Data, 2009-2010; Institute of Education Sciences database of SIG-awarded schools. 

Note: Data from 2008-2009 were used for states and districts with data missing in 2009-2010. Data from 

2007-2008 were used for states and districts with data missing in both 2009-2010 and 2008-2009. 
Data from 2009-2010 were used whenever possible because that was the school year just before the 
first year of implementation of the ARRA-funded SIG intervention models. Percentages of students are 
unweighted state-level and district-level averages. The column for all states includes data for 50 states 
and the District of Columbia. The column for districts in the U.S. with at least one school implementing a 
SIG-funded intervention model include data for districts in 49 states and the District of Columbia 
because the database of SIG-awarded schools does not include information for Hawaii. The 
percentages of U.S. districts with at least one school implementing a SIG-funded intervention model are 
based on schools’ planned implementation as of 2009-2010 for cohort 1 grantees (which received 
awards in 2010) and as of 2010-2011 for cohort 2 grantees (which received awards in 2011) and 
include only Tier I and II schools (because Tier III schools that received SIG awards did not have to use 
those funds to implement a school intervention model). Two study districts were each composed of two 
districts located within a larger school system. For each of these districts, data for the two districts have 
been combined in the above analyses. 

* Significantly different from districts in the U.S. with at least one school implementing a SIG-funded intervention 
model at the 0.05 level, two-tailed test. 
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Exhibit 1. SIG Intervention Models as Described by the U.S. Department of Education SIG Guidance (2012) 

I. Turnaround Model 


A turnaround model is one in which a local education agency (LEA) must do the following: 

1) Replace the principal and grant the principal sufficient operational flexibility (including in staffing, 
calendars/time, and budgeting) to implement fully a comprehensive approach in order to substantially 
improve student achievement outcomes and increase high school graduation rates 

2) Use locally adopted competencies to measure the effectiveness of staff who can work within the turnaround 
environment to meet the needs of students: 

A. Screen all existing staff and rehire no more than 50 percent 

B. Select new staff: 

(1) Implement such strategies as financial incentives, increased opportunities for promotion and 
career growth, and more flexible work conditions that are designed to recruit, place, and retain 
staff with the skills necessary to meet the needs of the students in the turnaround school. 

(2) Provide staff with ongoing, high-quality, job-embedded professional development that is 
aligned with the school’s comprehensive instructional program and designed with school staff 
to ensure that they are equipped to facilitate effective teaching and learning and have the 
capacity to successfully implement school reform strategies. 

(3) Adopt a new governance structure, which may include, but is not limited to, requiring the 
school to report to a new “turnaround office” in the LEA or state education agency (SEA), hire 
a “turnaround leader” who reports directly to the superintendent or chief academic officer, or 
enter into a multiyear contract with the LEA or SEA to obtain added flexibility in exchange for 
greater accountability. 

(4) Use data to identify and implement an instructional program that is research -based and 
vertically aligned from one grade to the next as well as aligned with state academic standards. 

(5) Promote the continuous use of student data (such as from formative, interim, and summative 
assessments) to inform and differentiate instruction in order to meet the academic needs of 
individual students. 

(6) Establish schedules and implement strategies that provide increased learning time. 

(7) Provide appropriate social-emotional and community-oriented services and supports for 
students. 

(U.S. Department of Education, 2012, pp. 27-28) 

II. Restart Model 

A restart model is one in which an LEA converts a school or closes and reopens a school under a charter school 
operator, a charter management organization (CMO), or an education management organization (EMO) that has 
been selected through a rigorous review process. A restart model must enroll, within the grades it serves, any former 
student who wishes to attend the school (see C-6). 

(U.S. Department of Education, 2012, p. 31) 

III. Closure Model 

School closure occurs when an LEA closes a school and enrolls the students who attended that school in other 
schools in the LEA that are higher achieving. These other schools should be within reasonable proximity to the closed 
school and may include, but are not limited to, charter schools or new schools for which achievement data are not yet 
available. 

(U.S. Department of Education, 2012, p. 34) 

IV. Transformation Model 

An LEA implementing a transformation model must: 

1 ) Replace the principal who led the school prior to commencement of the transformation model. 

2) Use rigorous, transparent, and equitable evaluation systems for teachers and principals that — 
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A. Take into account data on student growth as a significant factor as weii as other factors, such as 
muitipie observation-based assessments of performance and ongoing coiiections of professionai 
practice refiective of student achievement and increased high schooi graduation rates. 

B. Are designed and deveioped with teacher and principai invoivement. 

3) identify and reward schooi ieaders, teachers, and other staff who, in impiementing this modei, have 
increased student achievement and high schooi graduation rates and identify and remove those who, after 
ampie opportunities have been provided for them to improve their professionai practice, have not done so. 

4) Provide staff with ongoing, high-quaiity, job-embedded professionai deveiopment that is aiigned with the 
school’s comprehensive instructionai program and designed with schooi staff to ensure they are equipped to 
faciiitate effective teaching and iearning and have the capacity to successfuliy impiement schooi reform 
strategies. 

5) impiement such strategies as financiai incentives, increased opportunities for promotion and career growth, 
and more fiexibie work conditions that are designed to recruit, piace, and retain staff with the skiils 
necessary to meet the needs of the students in a transformation modei. 

(U.S. Department of Education, 2012, pp. 27-28) 

Source: U.S. Department of Education. “Guidance on Fiscal Year 2010 School Improvement Grants Under 

Section 1003(g) of the Elementary and Secondary Education Act of 1965.” Washington, DC: Office of 
Eiementary and Secondary Education, U.S. Department of Education, 2012. Avaiiabie at 
http://www2.ed.gov/programs/sif/ sigguidance05242010.pdf. Accessed May 30, 2014. 
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Table 2. SIG Funding Awarded in 2010 and Number of Schools Implementing Each Intervention Model 




School Intervention Model 



Transformation 

Turnaround 

Restart 

Closure 

Tier III 
Strategies® 

Number of Schools Implementing Each Intervention Model 

Tier 1 

354 

138 

24 

8 

0 

Tier II 

255 

40 

9 

8 

0 

Tier III 

14 

0 

0 

0 

403 

Total 

623 

178 

33 

16 

403 

Distribution of Award Amounts (Over Three Years) 

10th Percentile 

$942,892 

$1,236,632 

$1,187,500 

$31,935 

$60,190 

50th Percentile 

$2,100,000 

$2,684,490 

$2,167,965 

$50,000 

$300,000 

90th Percentile 

$5,114,190 

$5,190,000 

$5,490,491 

$254,323 

$900,405 


Source: Institute of Education Sciences database of SIG-awarded schools; Hurlburt et al. (2011). 

Note: The SIG awards summarized in this table are from the round of state applications due to the U.S. 

Department of Education on February 8, 2010. The award amount percentiles are based on the total 
award amount per school. The maximum award amount was $2 million per year for three years (or $6 
million in total over three years). 

® Tier III strategies refer to all school improvement strategies adopted by SIG-awarded Tier III schools. Federal rules 
did not require Tier III schools to implement one of the four school intervention models. 
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Table 3. Eligibility Requirements for Implementing SIG-Funded Intervention Models^ 


Original Tier Definitions 

Tier I Any school receiving Title I funds in 
improvement, corrective action, or 
restructuring that — 

(i) is among the lowest-achieving 5 percent of Title 
I schools in improvement, corrective action, or 
restructuring or the lowest-achieving five Title I 
schools In improvement, corrective action, or 
restructuring in the state, whichever number of 
schools Is greater; or 

(il) Is a high school that has had a graduation rate 
that is less than 60 percent over a number of 
years 

Tier II Any secondary school that is eligible for, but 
does not receive. Title I funds that — 

(i) is among the lowest-achieving 5 percent of 
secondary schools or the lowest-achieving five 
secondary schools In the state that are eligible 
for, but do not receive. Title I funds, whichever 
number of schools Is greater; or 

(il) Is a high school that has had a graduation rate 
that is less than 60 percent over a number of 
years. 

Tier III Schools receiving Title I funds in improvement, 
corrective action, or restructuring that are not 
in Tier I. 


Source: U.S. Department of Education. 


Expanded Tier Definitions 

Title l-eligible elementary schools*^ that are no 
higher achieving than the highest-achieving 
school that meets the original Tier I definition 
and that are: 

• in the bottom 20 percent of all schools In the 
state based on proficiency rates; or 

• have not made AYR for two consecutive years. 


Title l-eligible secondary schools*^ that are 
(1) no higher achieving than the highest- 
achieving school that meets the original Tier II 
definition or (2) high schools that have had a 
graduation rate of less than 60 percent over a 
number of years and that are: 

• in the bottom 20 percent of all schools In the 
state based on proficiency rates; or 

• have not made AYR for two consecutive years. 

Title l-eligible schools'^ that do not meet the 
requirements to be in Tier I or Tier II and that 
are: 

• in the bottom 20 percent of all schools In the 
state based on proficiency rates; or 

• have not made AYR for two years. 


®The original tier definitions were published in the Federal Register on December 10, 2009. The expanded tier 
definitions were published in the Appropriations Act on December 16, 2009. 

'^Title l-eligible schools include all schools eligible to receive Title l-funds, including both those that do and do not 
actually receive the funds. 
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Table 4. Opportunities to Implement an ROD Based on SIG Eligibility Tier Definitions^ 


Opportunity 

Treatment Group 

Comparison Group 

Assignment Variable'^ 

1 

Original Tier 1 Elementary 

Original Tier III Elementary 

Achievement 

2 

Original Tier 1 Secondary 

Original Tier III Secondary 

Achievement 

3 

Original Tier 1 Secondary 

Original Tier III Secondary 

Graduation Rate 

4 

Original Tier II Secondary 

Original Tier II Secondary, 
but above the cutoff‘s 

Achievement 

5 

Original Tier II Secondary 

Original Tier II Secondary, 
but above the cutoff'^ 

Graduation Rate 

6 

Expanded Tier 1 Elementary 

Expanded Tier III Elementary 

Achievement 

7 

Expanded Tier II Secondary 

Expanded Tier III Secondary 

Achievement 

8 

Expanded Tier II Secondary 

Expanded Tier III Secondary 

Graduation Rate 


Source: State administrative records. 

®in an RDD, the assignment variabie is the variabie used to assign units to the intervention. 

“^The originai tiers are those based on the definitions pubiished in the Federai Register on 
December 10, 2009. The expanded tiers are those based on the definitions pubiished in the Appropriations Act on 
December 16, 2009. 

'^Schoois that are eligibie for, but do not receive, Titie I funding and are above the 5 percent achievement cutoff. 
'^Schoois that are eiigibie for, but do not receive, Titie I funding and are above the 60 percent graduation rate cutoff. 
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Table 5. Comparing IK and Cross-Validation (CV) 


School 

Sample 

Size 

Failure % 

Mean of 
Absolute 
Value of Bias 

Impact Standard 
Deviation 

Mean Square Error 

IK 

CV 


IK 

CV 

IK 

CV 

IK 

CV 

Mean Across 31 Data Generating Processes 

10 

20 

44 


0.19 

7.92 

2.35 

185.7 

5.79 

208161 

20 

0 

2 


0.07 

0.17 

2.38 

3.64 

5.98 

19.9 

50 

0 

0 


0.02 

0.09 

0.86 

0.55 

0.76 

0.31 

500 

0 

0 


0.01 

0.05 

0.22 

0.17 

0.05 

0.03 

Percentage of Data Generating Processes for Which Each Statistic Is Smaller 
Than for Competing Bandwidth Method 

10 

- 

- 


87 

13 

100 

0 

100 

0 

20 

- 

- 


87 

13 

65 

35 

65 

35 

50 

- 

- 


94 

6 

0 

100 

0 

100 

500 

- 

- 


100 

0 

0 

100 

0 

100 


Source: Monte Carlo simulations, 1 ,000 replications. 

Note: We assessed the performance of the analytic methods described above, and compared the 

performance of different bandwidth selection methods, using Monte Carlo simulations that mimic study 
data observed from the year before SIG had an opportunity to affect outcomes. Thus, the simulations 
were conducted under the null hypothesis of no true impact, using a data generating process (DGP) 
that mimics the relationships between outcomes and assignment variables observed in our actual data. 
To estimate these relationships, we ran quartic regressions of test scores on assignment variables for 
every grade and ROD opportunity for which we have data. These regressions were run as mixed effects 
models to account for clustering of students within unique values of the assignment variable. Test 
scores were converted to z-scores; assignment variables were centered at the cutoff and divided by the 
standard deviation. We recorded the coefficient estimates from each regression, the intraclass 
correlation, and the cluster-level regression R2; these parameter estimates from each regression 
constituted a potential DGP for our simulations. We limited the simulations to 31 DGPs (shown in Table 
6 below) that were stable (i.e., had ICC and R2 estimates between 0 and 1) and had a strong 
relationship between the outcome and assignment variable (meaning that they were more prone to 
creating bias in RDD impact estimates). Error terms and assignment variables were generated as 
standard normal. For each of the 31 DGPs, and for 4 different sample sizes, we ran simulations in 
which impacts were calculated using bandwidths selected by both IK and cross-validation. For larger 
sample sizes (50 or 500 schools) we see that the IK method results in impact estimates with smaller 
bias but greater variance and greater mean square error. For smaller sample sizes (10 or 20 schools), 
we see that the IK method has smaller bias, smaller variance, a smaller mean square error, and a lower 
failure rate (the failure rate is the proportion of Monte Carlo replications where it was not possible to 
estimate a valid impact). The sites in our study are more likely to have sample sizes in the range of 10 
to 20 schools; IK outperformed cross-validation by all metrics in those cases. 
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Table 6. Regression Estimates Using Data from 31 Grade/RDD Opportunity Combinations 


Combination 

bO 

b1 

b2 

b3 

b4 

ICC 


1 

-0.531 

0.085 

0.024 

0.031 

-0.011 

0.101 

0.243 

2 

-0.454 

0.08 

0.004 

0.034 

-0.008 

0.095 

0.386 

3 

-0.439 

0.105 

-0.065 

0.024 

0.004 

0.097 

0.511 

4* 

-0.474 

0.057 

0.006 

0.025 

-0.017 

0.081 

0.455 

5 

-0.436 

0.045 

-0.033 

0.026 

-0.008 

0.103 

0.28 

6* 

-0.552 

-0.177 

-0.327 

0.549 

-0.142 

0.088 

0.55 

7 

-1.021 

-0.074 

-0.026 

0.243 

-0.073 

0.129 

0.447 

8* 

-0.787 

0.332 

0.361 

-0.233 

0.032 

0.1 

0.553 

9 

-0.672 

-1.645 

3.214 

-1.547 

0.224 

0.101 

0.549 

10 

-1.062 

0.252 

-0.089 

-0.091 

0.049 

0.109 

0.403 

11 

-0.729 

0.324 

-0.557 

0.369 

-0.064 

0.063 

0.569 

12 

-0.285 

-0.55 

1.025 

0.772 

-0.9 

0.075 

0.286 

13 

-0.525 

2.073 

-12.808 

15.198 

-4.996 

0.122 

0.241 

14 

-0.812 

-0.212 

-0.07 

0.244 

0.102 

0.067 

0.23 

15* 

-0.768 

0.066 

0.021 

0.177 

-0.061 

0.143 

0.391 

16 

-0.852 

-0.257 

-0.225 

0.335 

-0.088 

0.058 

0.27 

17 

-1.196 

3.127 

-6.21 

4.77 

-1.15 

0.126 

0.516 

18 

-1.055 

0.938 

0.026 

-1.23 

0.582 

0.118 

0.305 

19 

-1.035 

-0.149 

0.266 

-0.054 

0.002 

0.148 

0.484 

20 

-0.918 

0.073 

0.011 

0.024 

-0.004 

0.129 

0.529 

21 

-0.985 

0.146 

-0.068 

0.054 

-0.008 

0.128 

0.518 

22 

-0.707 

0.326 

-0.385 

0.243 

-0.044 

0.137 

0.278 

23* 

-0.795 

0.459 

-0.057 

-0.169 

0.03 

0.102 

0.358 

24 

-0.418 

-0.298 

0.046 

0.275 

-0.096 

0.061 

0.452 

25 

-0.441 

-0.122 

0.158 

0.302 

-0.143 

0.117 

0.45 

26 

-0.557 

0.044 

0.109 

-0.046 

0.013 

0.094 

0.512 

27 

-0.976 

-0.717 

-0.339 

1.745 

-0.687 

0.123 

0.406 

28 

-0.914 

-0.121 

0.086 

0.04 

-0.01 

0.126 

0.27 

29 

-0.983 

-0.048 

0.051 

0.017 

-0.003 

0.123 

0.231 

30 

-1.059 

0.209 

-0.173 

0.021 

0.012 

0.107 

0.237 

31 

-1.479 

0.004 

-0.168 

0.062 

0.03 

0.062 

0.275 


Source: Regression estimates using data from the year prior to SiG impact anaiysis. 

Note: The regression equation is Y = bO + b1*X + b2*X^ + b3*X^ + b4*X'', where Y is a test score on a state 

standardized assessment and X is the RDD assignment variable. 

*These five sets of coefficient estimates yielded the largest bias using the IK bandwidth selection method. 
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Table 7. Comparing Local, Hybrid, and Study-Wide IK Methods 


Sample Size 

Faiiure % 

Bias 

Impact Standard Deviation 

Schools 

Sites 

Locai 

Hybrid Study-wide 

Locai 

Hybrid Study-wide 

Locai 

Hybrid Study-wide 

100 

5 

28 

44 5 

-0.02 

0.00 0.00 

1.19 

1.07 0.76 

200 

5 

4 

9 5 

0.00 

0.00 0.00 

0.70 

0.62 0.59 


Source: Monte Carlo simulations, 10,000 replications, results shown for Fuzzy Frontier impact estimate 

(Reardon and Robinson, 2010). 


Note: From among the 31 data generating processes (DGPs) shown in Table 6 above, we identified the 5 that 

resulted in the largest bias using IK (these are indicated in Table 6). We then constructed a simulated 
study consisting of 5 sites, each of which involves two assignment variables that influence the outcome 
through different combinations of those 5 DGPs. Using the three different bandwidth selection methods 
described above (local IK, study-wide IK, and hybrid IK), we calculated a study-wide impact using the 
Fuzzy Frontier method (Reardon and Robinson, 2010). We focused on the 5 DGPs with the largest site- 
level bias both to be conservative with respect to bias and to limit the computational cost of these 
simulations. We concluded that the study-wide IK method is the most effective because it yields no bias, 
has the smallest variance, and has the lowest failure rate with small sample sizes. 
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